Summary of Changes

Hello @sm745052, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the SourceDB to Spanner migration pipelines by automating the JDBC fetch size configuration. Previously, the fetch size might have been a fixed value or manually configured, potentially leading to suboptimal performance or resource issues. The new implementation dynamically calculates an appropriate fetch size by considering the Dataflow worker's machine type (CPU and memory) and the estimated size of rows being read from the source database. This automation aims to improve the efficiency and stability of data transfer, adapting to varying data characteristics and worker configurations.

Highlights

• Automated JDBC Fetch Size Calculation: Introduces a new FetchSizeCalculator to dynamically determine the optimal JDBC fetch size for data reads, based on Dataflow worker machine type (CPU and memory) and estimated row size from the source database.
• Enhanced Row Size Estimation: Extends the DialectAdapter and JdbcValueMappingsProvider interfaces with methods for estimating the size of a row for various database column types. Concrete implementations for MySQL and PostgreSQL are updated to provide these detailed size estimations.
• Centralized Dataflow Worker Machine Type Utility: Replaces the previous DataflowWorkerMachineTypeValidator with a more comprehensive DataflowWorkerMachineTypeUtils. This new utility can retrieve worker memory and core information directly from the Compute Engine API, cache it for efficiency, and perform machine specification validations.
• Seamless Integration and Refactoring: The automated fetch size calculation is integrated into the JdbcIoWrapper used by SourceDB to Spanner migration templates. This involves updating configuration classes (JdbcIOWrapperConfig, TableConfig, SourceTableSchema) and refactoring references to the new machine type utility across multiple template files.

Codecov Report

❌ Patch coverage is 85.80645% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.13%. Comparing base (6ae6430) to head (d88e2d2).

@@             Coverage Diff              @@
##               main    #3260      +/-   ##
============================================
+ Coverage     51.05%   51.13%   +0.08%     
- Complexity     5172     5596     +424     
============================================
  Files           978      980       +2     
  Lines         60468    60635     +167     
  Branches       6633     6670      +37     
============================================
+ Hits          30871    31008     +137     
- Misses        27431    27441      +10     
- Partials       2166     2186      +20